Overview of ABS Quality Program 
November 2008 


One of the ways the Australian Bureau of Statistics (ABS) upholds its credibility in the 
community is through dedicated adherence to maintaining the quality of all aspects of work 
and at the same time looking for ways to improve current practices. 


The ABS Making Quality Visible (MQV) project is a way in which the ABS has actively taken 
steps to formalise the process of maintaining quality. The Methodology and Data 
Management Division (MDMD) has a dedicated resource managing this project, along with 
the support of managers throughout the division who actively promote quality initiatives 
within their work programs and other areas in the ABS. 


This paper provides a brief overview of the quality initiatives that the ABS is currently 
undertaking along with a status update of progress. 


Quality initiatives related to 'Processes' 


Quality Gates (QGs) 


Quality gates (QGs) are a tool that are designed to improve the early detection and 
treatment of errors within a processing cycle. They are check points at significant stages 
within the cycle where an assessment is made of the quality before proceeding to the next 
stage of the cycle. The assessments can be either of a quantitative or qualitative nature. 
Each quality gate includes tolerance levels, roles and responsibilities of areas involved in 
regards to the quality gate, along with the action required at each gate. 


There has been wide acceptance of the concept of quality gates throughout the ABS and a 
number of areas are beginning to think about their processes and how quality gates may fit. 
Progress has been slow in the implementation of quality gates throughout the ABS. Part of 
the reason behind this is the initial development of quality gates requires time to flesh out 
the placement of the gates and acceptable range of results expected from the gates. 
However, quality gates will provide efficiency gains in the long term by identifying issues 
earlier than otherwise would have occurred. 


Quality Infrastructure System (QIS) 


The Quality Infrastructure System (QIS) is a repository which links to other corporately 
approved systems to allow the capture and storage of survey information to aide in the 
quality assurance of the entire end to end process. 


Monitoring of particular aspects of a survey, whether it be response rate, relative standard 
errors, data items etcetera, is available within the cycle, at the end of a cycle and with 
comparisons over time, through the use of the QIS. This ability enables the implementation 
of quality assurance processes, such as quality gates, to be monitored more easily. 


The automation of the loading of information into the repository means that survey areas, 
once loaded, can obtain up-to-date information throughout their process by logging into the 
QIS. 


The QIS has the ability to display survey information in various output forms, such as tabular 
and graphical. With the addition of Notes 8, future developments into widgets on desktops, 
such as dashboard type information for Senior Managers, will be investigated. 


Work is being undertaken this financial year to improve already existing functionality in QIS 
in order to promote the System more widely throughout the ABS as a key quality assurance 
tool. 


Clearance documentation 


Clearance documentation is management information that is presented at the end of the 
processing cycle before the release of a statistical product. This information focusses on the 
quality of the data for the current cycle and any changes to that quality over time. Issues 
that occurred during the cycle are presented in this document. For example unusual 
movements compared to the previous cycle will be highlighted in this document along with 
an explanation for why these unanticipated movements are correct. Explanations may 
include real world impacts that have affected the data. 


The ABS has a standard Clearance Documentation template for Economics collections and 
work is underway on a similar standard template for Social collections. 


Quality of time series 


The Time Series Analysis (TSA) area of the ABS has progressed the development of 
training courses. The first course developed - Understanding Time Series in the ABS - has 
been piloted in central office (CO) and the New South Wales (NSW) office. 


A second national statistical training institute (NSTI) training module which focuses on the 
quality assurance of seasonally adjusted and trend estimates produced during processing 
cycles is under development. This module is more process-oriented and will focus on how 
to check and interpret the log file warning messages generated from running concurrent 
seasonal adjustment each cycle. Improvements to these log files have also been made, 
including an overall summary of the cycle, and will be released into production later this 
year. 


Work is also progressing on developing recommended content for clearance documentation 
relating to seasonally adjusted and trend estimates. This work, along with the NSTI training 
course and log file warning messages, is designed to increase the capability of client areas 
to quality assure their processes and to increase their understanding of the time series they 
are analysing. 


Internally, TSA has also implemented quality gates for the Annual Seasonal Reanalyses 
(ASR) processes. These gates aim to ensure that all aspects of an ASR are completed 
thoroughly and consistently across collections. 


Quality reviews 


Quality reviews are a new concept within the ABS that are being piloted in 2008. The 
concept of these reviews was endorsed by senior managers of the ABS on the 30 October 
2007. The concept for quality reviews is heavily modelled on a similar program of quality 
reviews in Statistics Sweden, and on the Gateway Review process administered by 
Department of Finance and Administration (DoFA) for large IT projects. 


Quality reviews are an intensive review of a collection or a process, and are conducted over 
a short period of time (usually five working days). The quality review is conducted by a team 
of three independent reviewers from methodological, system and operational backgrounds 


and the findings of the review are provided to the area for which the review is being 
conducted, so that they can improve their processes where possible. 


The first pilot review was conducted for the confidentiality procedures for International Trade 
statistics in January 2008. The second review was conducted in May on the Retail Trade 
survey. A third review will be scheduled shortly, most likely for a household collection or 
some other area of social statistics. 


The pilot tests have revealed some issues to be addressed in the pilot quality review 
process, such as: clearer definition of the scope of the review; provision of a generic 
timetable for the reviewers to assist in the planning of the five days; the level of 
confidentiality that is appropriate for the final review reports; and a way of achieving a list of 
available panel reviewers, especially for emergency reviews. 


It is intended that an evaluation report of the pilot quality reviews and recommendations will 
be presented to senior managers within a month of the third and final pilot review being 
completed. 


Internal Audits 


Internal audits are one of the ways the ABS ensures that our internal practices are efficient 
and correct. The ABS is currently conducting a top down review of our systems to identify 
areas of weaknesses. Areas of possible weakness identified from this review will undergo a 
more vigorous audit called a Quality Review (discussed above). 


Quality initiatives related to 'Capability' 


Quality training 


MDMD is developing a suite of three NSTI courses to develop skills and knowledge relating 
to the making quality visible vision. These are Quality Concepts and Frameworks; Managing 
Statistical Risk; and Reporting Quality in the ABS. It is expected that these three courses 
will replace the current Data Quality in the ABS course. 


Quality Concepts and Frameworks explains why the ABS has such a vested interest in 
maintaining quality and examines the different tools that have been developed to help with 
this direction. The course was piloted on the 31st of October 2008 in Central Office, with 
three attendees from Statistics New Zealand. Future presentations of this course will occur 
as demand dictates. 


Managing Statistical Risk will discuss in detail tools which can be used by areas in the ABS 
to help with identifying statistical issues in the end to end process, and how to deal with any 
quality incidents once identified. Quality gates are one of the tools that will be discussed in 
detail along with documentation that is available for quality management. The course is 
scheduled for release in early 2009. 


Reporting Quality in the ABS examines both internal and external data needs in the context 
of the seven dimensions of the ABS Data Quality Framework. The course primarily helps 
participants gain a better understanding of the creation of the content of quality declarations 
(discussed further below) which use the ABS Data Quality Framework. As quality initiatives 
evolve this course will be updated to reflect the current needs of staff. 


Quality Manual 


This manual is a central repository for all staff in the ABS. It contains information on a wide 
range of topics relating to management of the end to end cycle. It is an evolving document 
as more chapters will be added as developed. 


The quality manual contains information on: 

the ABS Data Quality Framework; 

the Quality Declaration User Guide (a how to, and content document); 
Quality Gates; 

Quality Incident Response Plan; 

Statistics Sourced from Administrative Data; and 

A Template for quality assurance for change processes. 


The ABS Data Quality Framework was developed from two international frameworks - 
Statistics Canada and Eurostat, to come up with the seven dimensions of data quality. 
These dimensions are Institutional Environment, Relevance, Timeliness, Accuracy, 
Coherence, Interpretability and Accessibility. These seven dimensions are used to help 
define the content of Quality Declarations (discussed below). Further information on the 
content of this framework can be found in the ABS Annual Report which is on the ABS 
website. 


The Quality Gates section of the Quality Manual contains information on what a quality gate 
is, how it fits in the end to end process and how to develop them. 


The Quality Incident Response Plan (QIRP) is a plan for resolving serious doubts about key 
statistical results. The QIRP approach is designed to provide a quick and rigorous high level 
response to an identified serious statistical problem and is based upon the formation of a 
dedicated ad hoc multidisciplinary team to investigate and resolve the issue. The process is 
pragmatic and focuses in quickly on the events and issues that really matter. This manual 
contains information on the practicalities of a QIRP, what to do and when, and recognising a 
QIRP in the first instance. 


The Statistics Sourced from Administrative Data chapter looks at the management of the 
quality of statistical outputs produced from administrative data from the initial acquisition 
phase to final release of data. 


The Template for quality assurance for change processes provides easy to follow 
instructions for monitoring any known change factors in a process. The template goes 
through the steps of identifying the known changes and then creating quality assurance 
procedures to manage these changes. This includes identifying risks, measures and people 
responsible for the monitoring. 


Quality Assistant 
The Quality Assistant is a database that provides links to useful information regarding quality 


initiatives in the ABS. It contains links to manuals, international papers and ABS specific 
papers. All staff are able to access the Quality Assistant from their desktops. 


Quality Framework for Assessing Model-based Estimates 


A quality framework for assessing model-based estimates was presented to the 
Methodology Advisory Committee (MAC) in June 2006. This framework contains the 


following dimensions: Relevance, Accuracy, Plausibility, Consistency, Robustness, 
Interpretability, Transparency, Endogeneity, Sustainability and Cost Efficiency. Depending 
on the assumptions underlying the model the importance of each of these dimensions will 
vary when assessing the quality of model-based estimates. It was noted in the MAC paper 
that the criteria set out in the framework might not be exhaustive and that depending on 
specific circumstances other factors may need to be taken into account in the quality 
assessment. Further work on the development of this framework will continue based on 
advice received from the MAC. 


Quality initiatives related to Outputs’ 


Quality Declarations (QDs) 


Quality declarations are short, sharp statements describing the quality or fitness-for-purpose 
of a statistical product that are designed to operate effectively in a web-based environment. 
They are succinct and concise pieces of information designed to convey some of the quality 
aspects of a particular collection using the seven dimensions of the ABS data quality 
framework. The presentation of information is layered so that casual users obtain a brief 
headline indication of quality, whilst users who are interested in more detail can easily 
access it. The implementation of QDs was endorsed in May 2007 and the roll-out to began 
in 2008 to the ABS website. 


The introduction of QDs to specific ABS products increases in 2009 to include compendiums 
and confidentialised unit record file (CURF) releases. 


Quality Page 


Discussion has been held recently on the idea of developing a ‘Quality’ theme page on the 
ABS website which would provide information to users about the quality initiatives that the 
ABS uses in order to provide high quality data. 


Endeavours to release the ABS Data Quality Framework plus other informative 
documentation on quality assurance practices has started, with releases to the ABS website 
expected from early 2009. 


